This is a rare edge case, but can be consistently triggered in some development setups.

Requirements to reproduce:

1. Postgres storage.
2. Sync that ignores specific rows (i.e. the data is replicated to current_data, but not in bucket_data / synced to clients). This can be done by adding a data query with where false in the sync rules.
3. Start with one or more records for the above table, and no other records.
4. Sync one client.
5. Now add a row that does get synced.
6. The client will now go into a repeated "Checksum failed for ..." loop.

The issue is in how op_id_sequence is initialized in postgres storage. We use select last_value from op_id_sequence to get the op_id for a checkpoint, and select nextval('op_id_sequence') for each op_id in bucket_data. last_value will generally return the largest op_id value in bucket_data, and nextval will always generate a larger op_id. However, if we haven't generated any op_id yet, and generate the first checkpoint, then we get the case that last_value is 1 (used for the first checkpoint). Then for the next checkpoint, nextval gives 1, giving another checkpoint with the same op_id. Together with caching of checksums, this results in a op_id conflict that gives the "Checksum failed" loop.

To see what's going on, run curl http://localhost:8080/sync/stream -X POST -H "Authorization: Bearer <token>. You should see no data initially:

{"checkpoint":{"last_op_id":"1","buckets":[{"bucket":"global_data[]","checksum":0,"count":0,"priority":3}]}}
{"checkpoint_complete":{"last_op_id":"1"}}

Then add a record to sync. The above stream would not get a new checkpoint, while re-running will give this:

{"checkpoint":{"last_op_id":"1","buckets":[{"bucket":"global_data[]","checksum":0,"count":0,"priority":3}]}}
{"data":{"bucket":"global_data[]","after":"0","has_more":false,"data":[{"op_id":"1","op":"PUT","object_type":"test","object_id":"abc","checksum":287648289,"subkey":"...","data":{"id":"abc"}}],"next_after":"1"}}
{"checkpoint_complete":{"last_op_id":"1"}}

Note above:

1. last_op_id is still 1.
2. The checkpoint for the bucket still contains "checksum: 0" and "count: 0".
3. There is however a data line for that bucket.

This causes the checksums to fail on the client.

The issue specifically happens because the count and checksum for op_id 1 is cached and never refreshed. Even when more data is added, the op_id is incrementally updated, and still remains incorrect. The service recovers when it is restarted, clearing the cache.

However, the cache is not the core issue here - op_id 1 being re-used for two checkpoints is the issue.

The fix

One simple fix is to just call select nextval('op_id_sequence') once as part of the migrations. This initializes the sequence, avoiding this issue completely. However, this also changes the op_id sequence for all our test fixtures. Rather than updating them all, and accounting for differences between mongodb and postgres storage, this fix just checks is_called on the sequence, and defaults to 0 instead of 1 for that case.